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ABSTRACT 


Recent studies suggest that the number of students pursuing science, technology, engineering, and 
mathematics (STEM) degrees has been generally decreasing. An extensive body of research cites the 
lack of motivation and engagement in the learning process as a major underlying reason of this decline. 
It has been discussed that if properly implemented, instructional technology can enhance student 
engagement and the quality of learning. Therefore, the main goal of this research is to implement and 
assess the effectiveness of an augmented reality (AR)-based pedagogical tool on student learning. For 
this purpose, a model building design and assembly experiment with two separate (control and test) 
treatments were conducted in an undergraduate construction and civil engineering course. In each 
treatment, performance (with respect to three primary and three secondary measures) and workload 
data (with respect to six NASA TLX indicators) were collected and assessed. Both treatments were 
also videotaped for post-analysis and further observations of students’ performance. It was found 
that students in the test group (who used the AR tool) performed better than students in the control 
group with respect to certain (but not all) measures. In addition, test group students spent more time 
on collaboration, communication, and exchanging ideas. Overall, students ranked the effectiveness of 
the AR tool very high and stated that it has a good potential to transform traditional teaching methods. 

Key Words: Augmented reality; construction and civil engineering; building design. 

INTRODUCTION 

According to the National Academies Press (NAP), during the past two decades, the number of 
students pursuing bachelor’s degrees in science, technology, engineering and mathematic (STEM) 
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disciplines decreased by 18% in the United States (Augustine, 2007). Moreover, only 23% of college 
freshman students declare a STEM major and only 40% of those that choose STEM, receive a STEM 
degree by the end of their studies (Duncan, 2009). Recently, the United States ranked 17th amongst 
the developed countries in the proportion of college students receiving bachelor’s degrees in science 
and engineering (Augustine, 2007). These and several other statistics have motivated researchers 
to look for the underlying reasons for this decline. 

Some researchers discussed that the relatively high upfront monetary investment necessary to 
earn an engineering degree may be a setback to many students (Shin and Milton, 2008). However, 
this may not be necessarily true since figures show that the salary of a typical engineer is much 
higher than many other majors (Gardner, 1992). Some educators have argued that the decision 
to pursue a STEM major is based on two key factors: (1) personal capabilities and preparedness 
to succeed, and (2) desire to pursue that discipline. They believe that success in attracting more 
students into the STEM fields depends on how well the universities and institutions address both 
of these components (Cutler, 2012). Other researchers indicated that the problem is not attract¬ 
ing students into the STEM fields; rather it is retaining them there throughout their studies and 
engaging them in the learning process (Sonmatel, 2013; Lin, 2013; Kane, Pando, and Janardhnam, 
2013). According to the U.S. Department of Education statistics, the United States is graduating 
more visual-arts and performing-arts majors than engineers. The same study showed that China 
was graduating more English-speaking engineers than the United States does. One of the major 
differences between the United States and the developing countries is the existence of advanced 
technologies in people’s daily lives which could naturally result in a higher demand for technology 
integration in education as well (Olson and Riordan, 2012). Along these lines, outdated and poor 
teaching methods, disconnection between students and classroom technology, and lack of hands- 
on experiments are among important reasons that keep some students away from pursuing STEM 
disciplines (Chen and Soldner, 2014). 

Therefore, the overarching objective of this research is to find ways to facilitate the transforma¬ 
tion of difficult (and often boring) course topics into a more engaging and easy-to-understand 
learning experience. It is well known that integrating technology into higher education can comple¬ 
ment, supplement, and enhance the components common to any instructional model (Kent and 
McNergney, 1999; Bates and Poole, 2003). Along this line, some studies have concluded that the 
latest technology such as portable electronic devices (PEDs) (e.g. laptops, smartphones, and 
tablets) have become an integral part of many students’ learning toolbox. While some may argue 
that such tools can be a source of distraction (Roschelle, Patton, and Pea, 2002), they can also 
provide an opportunity for engaging students, if used properly (Khalid, Chin, and Nuhfer-Halten, 
2013). Previous researchers have illustrated how mobile technologies can be used to (a) facilitate 
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guided participation among undergraduate engineering students, and (b) teach graduate students 
in instructional technology to design for guided participation (Evans and Johri, 2008; Kim, Mims, 
and Holmes, 2006). 

Ultimately, the goal of all such research projects has been to enable educators to use technology- 
enhanced learning beyond just the desktop or classroom computers and towards creating value¬ 
adding links between information and communications technology (ICT) and other classroom 
activities (Amelink, Scales, and Tront, 2012). Even if such technologies are not yet user friendly and 
fully affordable, the pedagogy underlying these approaches can be used as a source for introducing 
ICT to students for teaching and learning purposes. One such technology that has been drawing 
significant attention during the past few years by instructors and educators is augmented reality 
(AR) visualization. Previous work conducted by the authors has demonstrated the high potential of 
using this technology to enhance the instructional methods in engineering and science (Shirazi and 
Behzadan, 2013). Other researchers have also investigated the potential of AR in their respective 
fields (Miyashita et al., 2008; Klopfer et al., 2005). Building upon the authors’ previous experience 
with the application of AR in educational and training contexts, the aim of the research presented 
in this paper is to investigate the extent to which AR can be used in large-scale classroom settings 
to enhance the learning quality and involve all students in class activities. As discussed later, the 
findings of this study can also support the effective use of AR in more interactive environments such 
as laboratories and hands-on sessions while the instructor may not be able to provide all individuals 
or groups with sufficient material (Stockton, 2012). 

Use of Augmented Reality in Construction and Civil Engineering Instruction 

As previously stated, the number of undergraduate students pursuing engineering disciplines in 
North America and Western European countries has been following a decreasing trend during past 
few years (Munce, 2013; Johnson and Jones, 2006). This decline was even more noticeable in construc¬ 
tion and civil engineering in comparison to other engineering disciplines (Nehdi, 2002; Yoder, 2011). 

As far as instructional models are concerned, most educators in construction and civil engineer¬ 
ing have to deal with several major problems. Among them, lack of practical experience to relate 
fundamentals to practice is considered as one of the most important issues (Koehn, 2004). Also, 
limited instruction time, lack of technology, and the use of traditional teaching methods are among 
other problems. Moreover, liability and safety issues, and adverse weather in some geographical 
locations have limited the number of field trips and jobsite visits students can make during the 
academic year (Shaaban, 2013). In the long term, this will negatively affect instructors’ ability to 
relate theoretical concepts to real world scenarios by showing students how abstract topics are 
implemented in real projects. 
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In addition, the nature of construction and civil engineering which deals with building things with 
steel, concrete, brick, wood, and mud with little or no support of ICTs has caused instructors to limit 
their use of technology in the classroom (Sacks and Barak, 2009). The problem arises when the 
expectations and ambitions of the new generation of technology savvy students is added to the mix. 
In fact, many students complain about the disconnection between how they perceive technology and 
how technology is in fact used in the classroom (Levin and Wadmany, 2006). It can thus be inferred 
that an effective technology-based instruction coupled with conventional teaching methods cannot 
only help students learn better and faster, but will also enable instructors to become more familiar 
with new technology-based educational innovations and help them enhance their teaching quality. 
However, finding the best technology that can fulfill both students’ and instructors’ expectations at 
the same time is not always a trivial task. 

The authors conducted an academic survey on 241 junior-level students (consisting of 15% female 
and 85% male) of civil, environmental, and construction engineering at the University of Central 
Florida (UCF) in 2012-13. Figure 1 shows the breakdown of participants’ academic majors. Results 
indicated that 92% of respondents identified themselves as visual learners. In particular, this group 
agreed to the statement that “they learn better when the instructor uses 2D/3D visualization or 
multimedia to teach abstract engineering and scientific topics” (Table 1). Moreover, 54% claimed 
that “they learn better while working in a collaborative setting (e.g. working in a team) where they 
can play a role in the learning process” (Table 1). 

As discussed earlier, several studies supported the positive effect of using PEDs on student learn¬ 
ing and engagement (Khalid, Chin, and Nuhfer-Halten, 2013; Anderson et al., 2003). However, it is 
clear that not all academic institutions and universities are financially capable of providing high- 
tech devices and equipment to students. Therefore, one major concern in this and similar studies 
is the issue of affordability. For this reason, survey respondents were also asked to indicate if they 
already own a technology-enabled device that can be readily used in the classroom; 93% declared 
that they own either a smartphone or a tablet device or both (Figure 2), and can easily use it in 
their daily activities. 



Statement 

Agree 

Neutral 

Disagree 

I learn better when the instructor uses 2D/3D visualization or multimedia to teach abstract 
engineering and scientific topics. 

92% 

6% 

2% 

I learn better while working in a collaborative setting (e.g. working in a team) where they 
can play a role in the learning process. 

54% 

29% 

17% 

Table 1. Breakdown of students’ responses to the survey statements. 
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The fact that most students identified themselves as visual learners coupled with the large popula¬ 
tion of students who have a mobile device in their possessions, motivated the authors to pursue the 
use of a visualization technology that can be conveniently integrated into common mobile computing 
platforms. Following the background survey results and through evaluating several options, mobile AR 
was finally selected as a suitable candidate for evaluating the pedagogical potential of ICTs in classroom 
settings. In a nutshell, AR displays computer generated virtual objects over the views of the real world 
(Azuma, 1997). Unlike virtual reality (VR), AR enhances the views of a real environment with more visual 
information rather than completely replacing it. Therefore, it can create an interactive interface to work 
with mixed views of real world and virtual objects. Although AR is a relatively new technology, several 
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studies have shown the effectiveness of AR in education and training when used together with traditional 
teaching methods (Billinghurst and Dunser, 2012; Kaufmann and Schmalstieg, 2003). 

More specifically to the area of this research, in construction and civil engineering, AR can facilitate 
the creation of virtual immersive jobsites and thus, potentially eliminate safety and health related 
problems that may otherwise occur when taking students to active jobsites (Xue et al., 2013; Hen¬ 
derson and Feiner, 2009). In addition, AR visualization naturally supports discovery-based learning 
as well as promotes social interactions (Yuen, Yaoyuneyong, and Johnson, 2011). 

Literature Review and Research Objective 

There have been several AR applications designed and implemented in construction and civil 
engineering. For instance, Webster et al. (1996) presented his work in developing AR systems to 
improve methods for the construction and renovation of architectural structures. Other researchers 
worked on 4-dimentional (4D) AR models for automating construction progress and data collection 
(Golparvar-Fard, Peha-Mora, and Savarese, 2009). Some investigated the application of the global 
positioning system (GPS) and 3 degree-of-freedom (3-DOF) angular tracking to address the registra¬ 
tion problem during interactive visualization of construction graphics in outdoor AR environments 
(Behzadan and Kamat, 2007). Visualization techniques for field construction and outdoor AR are 
other topics recently presented in construction and civil engineering (Behzadan and Kamat, 2005; 
Kamat et al., 2010). Dunston (2009) discussed a number of technical issues associated with the 
application of AR systems in construction including displays, tracking, and calibration. Chen and 
Wang (2010) presented a framework for multi-disciplinary collaboration, discussed that tangible AR 
is a suitable system for design collaboration, and illustrated the need for integrating tangible user 
interfaces (TUIs) and AR systems. AR and other advanced visualization applications have been also 
used for educational purposes in construction training and sustainable design (Messner and Horman, 
2003; Vassigh, 2008). Moreover, it has been illustrated that VR and AR assistance in an assembly 
task could be helpful and increase productivity (Boud et al., 1999; Tang et al., 2003). Different AR 
applications enabled engineers to design and plan a product assembly and its assembly sequence 
through manipulating virtual prototypes in a real assembly workplace (Ong, Pang, and Nee, 2007). 

To the authors' best knowledge, most of the previous efforts implemented AR and tested its 
functionality without any further assessment. In contrast, the goal of this research is not only to 
design and implement an AR application for construction and civil engineering education but also 
to test its applicability in a real university course to evaluate its real effect on students’ learning 
and communication skills. Assessing an effective AR system requires the consideration of two 
critical concepts of (1) human factors for designing and developing the visualization tool as well as 
supporting the collaborative work, and (2) developing a technology infrastructure to "augment” 
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the control and inspection interfaces for direct access to the project which is spatially referenced 
and displayed. (Dunston and Wang, 2005; Wang and Dunston, 2006). In particular and consider¬ 
ing the recent advances and limitations, in this research, the effectiveness of using AR instructions 
in a building design and assembly task as part of an engineering course is tested. In addition to 
the technology design and implementation, and in order to systematically validate the designed 
pedagogical methodology, students’ performance data and their suggestions and evaluations were 
also collected and analyzed. The following Sections provide a detailed description of the technical 
and pedagogical aspects of the designed methodology, validation, and results of the experiment. 


MOBILE AUGMENTED REALITY SYSTEM DESIGN 


In this study, the authors designed, implemented, and assessed an AR-based pedagogical tool to 
help students better engage in the learning process and to create an environment in which students 
are motivated to learn abstract construction and civil engineering topics in a more practical manner. 
For this purpose, an experiment with two separate treatments were designed and conducted to 
test the effectiveness of AR instructions in a building design and assembly task in an undergradu¬ 
ate engineering course. 

The AR platform used in this project was designed based on Junaio, an open-source web-based 
augmented reality experience language (AREL) programming environment (Madden, 2011). Junaio of¬ 
fers a free, web-based application programming interface (API) which enables users to access the AREL 
content and create various AR applications. The AREL package includes three different components: (1) 
the static extensible markup language (XML) to define the content and linkages, (2) the Javascript logic 
to define dynamic parts such as user interactions, and (3) the content itself which includes 3D objects, 
images, and other multimedia files. The source of the AREL is identified by a channel content uniform 
resource locator (URL). This URL delivers the AREL XML through the mobile application. Using this 
process, when a user scans a quick response (QR) code corresponding to a specific channel, a hypertext 
transfer protocol (HTTP) request will be sent to the server. The server will then forward the request to 
the channel content URL and responds to the request with a static or dynamic XML. This XML will be 
then forwarded to the user and enables him or her to receive desired content such as 3D models, im¬ 
ages, movies, or other multimedia. The sequence diagram of the user query process is shown in Figure 3. 

In this research, two scenarios were combined to design the final AR experiment. Each scenario 
was defined in a separate channel which was registered on an online server. In particular, a location- 
based and an image-based channel were created. Using the location-based channel, users could 
hold their mobile devices and look around to see the virtual objects at the positions of points of 
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interest (POIs). In the designed experiment, an AR instructor (avatar) was created using location- 
based channels. Students first scanned the QR code and then held their mobile devices towards 
the instructor avatar (placed on a specific POI in the classroom) to access a step-by-step guide on 
how to conduct the experiment. As illustrated in Figure 4, each step was shown as a thumbnail that 
could be selected by students. Each thumbnail was linked to a video describing the details of that 
step. Therefore, students could watch any part of the instructions at their own pace and for any 
number of times during the course of the experiment. 

In contrast to location-based channels, image-based channels are used to attach or “glue” virtual 
3D models and other multimedia to any real object. In the designed experiment, image-base chan¬ 
nels were used to attach 2D/3D virtual information to each model building element. In this case, 
students were able to receive information (e.g. material type, weight, cost, dimensions) augmented 
on top of each element of the model building by moving their mobile devices over the tracking im¬ 
age attached to that element and scanning that image, as shown in Figure 5. 


PEDAGOGICAL METHODOLOGY 


In order to evaluate its pedagogical impact, the designed mobile AR application was tested in real 
classroom settings. In particular, and to compare the combined effect of employing a virtual instruc¬ 
tor and delivering contextual information via an AR interface, students were asked to participate in 
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Figure 4. Students used location-based channel to receive step-by-step descriptions from 
a virtual avatar. 
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two separate model building design and assembly experimental treatments. In the first treatment, 
participants were provided with a traditional (print) manual that contained detailed instructions and 
design information, while in the second treatment, students used their mobile devices to receive 
instructions as well as design information from the designed AR application. More information about 
these treatments, their effects on student learning, and the final evaluation results are presented in 
the following Subsections. 

Participants and Group Management 

Participants were junior and senior level construction and civil engineering students who were en¬ 
rolled in CGN3700C (Civil Engineering Measurements) in Fall 2013. Sixty students participated in this 
experiment with an average age of 24. The experiment was built into the course as two stand-alone 
laboratory modules. Participants were not given any prior information regarding the details of the 
experiment and had no previous experience with AR in an educational context. This was necessary 
to make sure that ail students were at the same level of practical knowledge prior to the experiment. 

Students were divided into two control and test sessions each consisting of 30 people. Each group 
conducted the experiment separately to avoid possible influence on the performance of the other 
group. In each session, students worked in groups of three. Students in the control group deployed 
ordinary print manual instructions, and students in the test group took benefit of the designed 
AR application and virtual instructor. Moreover, since some researchers have found that gender is 
correlated with spatial ability (Lawton, 1994), groups consisted of either male or female students 
to also examine possible gender effects. 

Experiment Procedure 

As described earlier, two treatments were created and conducted in two separate sessions: 

• Session 1: Print manual treatment (control group) 

• Session 2: AR instructor treatment (test group) 

In each session, participants were first introduced to the overall goals of the experiment. Follow¬ 
ing this brief introduction, no additional description was provided and groups were asked to begin 
the experiment. In session 1 experiment, each group was given a print manual handbook that con¬ 
tained descriptions of steps to complete the model building design and assembly task. All necessary 
primary and secondary measurement data was also included in the manual. Students were asked 
to follow the manual to determine what they had to do and make necessary decisions (Figure 6). 

In session 2, on the other hand, each group was given a 2-page handout containing only two QR 
codes linked to the location-based channel (i.e. virtual instructor) and a third QR code that provided 
linkage to the image-based channel (i.e. design and performance information of building elements). 
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Figure 6. Each group in the control session used print manual instructions to design and 
assemble a mode! building. 


As shown in Figure 7 (a), a large cardboard cutout of an avatar was placed in one corner of the room. 
Students used their mobile devices (smartphones or tablets) to scan the first two QR codes and then 
turn towards the avatar cutout to watch the corresponding instructional videos. Next, they used their 
mobile devices to scan the third QR code and gain access to design and performance information 
of model building elements. As shown in Figure 7 (b), the information was visually overlaid on top 



(a) (b) 

Figure 7. Students in the test session (a) worked in groups and received instructions from 
the virtual avatar, and (b) scanned the tracking image attached to each building element to 
access information. 
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of each building element as soon as the tracking image attached to that element was scanned and 
detected by the camera of a mobile device. 

Experiment Design 

The goal of the experiment was to design and build a model structure that meets minimum per¬ 
formance criteria. Each group received a package of 60 elements that could be assembled into a 
variety of building shapes. These elements were divided into three different categories of columns, 
beams, and junctions and finishing. At the beginning of the experiment, each team was asked to 
use three labels provided in the package to sort all pieces into these three categories. In addition 
to having three different element types, elements were also grouped into three materials namely 
concrete, steel, or wood. This was to encourage students to select the elements carefully consider¬ 
ing both shape and material properties such that the final building performance would be optimal. 
As described earlier, information relevant to each element was provided either in the print manual 
for the control group, or through AR visualization for the test group. 

Students had to also follow certain rules. Any deviation from these rules would be considered a 
design error and could add a penalty to the group’s final score. 

The experiment consisted of six steps, as follows. Students became familiar with these steps by 
reading their print manual (for control group) or communicating with the virtual instructor (for test 
group): 

1. Project description: The goal of the experiment and timing requirement were presented in this 
step. Also, a sample final product (i.e. model building) was shown to give students a better 
idea of what is expected at the conclusion of the experiment. 

2. Sorting the elements: All 60 elements were divided into three different categories of (1) col¬ 
umns, (2) beams, and (3) junctions and finishing. Before any further action, students were asked 
to use three labels provided in their building kit to sort all pieces into these three categories. 
They were able to check information relevant to each element using the provided tables in the 
print manual (for control group) or by scanning the tracking image on each element (for test 
group). In this step, AR information delivery platform helped students (in test group) find such 
information much easier by scanning each element rather than having to separately looking 
up each element in the tables provided in print manuals (for control group). 

3. Rules and regulations: Different rules and regulations each group had to consider when design¬ 
ing and assembling their structure was presented in this step. The rules described how to use 
beams and columns in the building, dimension requirements, and unacceptable configurations. 

4. Assessment factors and goals: Each group had to evaluate their final product using 6 factors 
(3 primary and 3 secondary, as described later). Students were introduced to these factors, 
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and were told that all factors had the same weight, and the final score would be determined 
by comparing their design to those of other groups in their session. 

5. Material information: In addition to having three different element types (as explained in step 1), 
each element was assumed to be either made of concrete, steel, or wood material. The choice 
of material could affect the performance of the final product in different ways. The material 
information was presented in this step in printed tables (for control group) or using virtual 
tables and AR information delivery (for test group). 

6.Overall assessment and conclusion: Each group was also provided with an assessment table 
to fill out at the end of their design and assembly experiment. This helped them organize and 
present ali required information and check for missing items, or if any correction was needed 
in their product design to achieve a better outcome. 

The final model building of each group was evaluated based on 3 primary measures (namely 
building volume, number of elements, and completion time) and 3 secondary measures (namely 
building cost, embodied carbon, and fire resistance). The goal was to make a model building with a 
volume closest to 30,000 cm 3 in the least possible time while using the fewest number of elements. 
The final models had to be at minimum cost, and result in the least possible carbon footprint and 
maximum fire resistance. Each group was provided with supplementary tables to help calculate 
all primary and secondary measurement factors for their model building. The final ranking of each 
group was then calculated relative to the performance of all 20 groups. Detailed information about 
calculating the ranking of each group and the final results are provided in the following Sections. 

Workload Assessment 

In order to evaluate and compare the task load in the two experimental treatments, the NASA 
task load index (NASA TLX) was used as an assessment technique. This subjective, multidimen¬ 
sional assessment tool is used to measure workload estimates associated with a task (Hart and 
Staveland, 1988). It considers 6 subscales that represent somewhat independent clusters of vari¬ 
ables indicating workload. The first three subscales related to demands on a person are (1) mental 
demand, (2) physical demand, and (3) temporal demand. The next three subscales related to 
person-task interaction are (1) frustration, (2) effort, and (3) own performance (Cao et al., 2009). 
Table 2 contains a detailed description of these subscales. The NASA TLX method assumes that 
some combinations of these dimensions are likely to represent the workload experienced by most 
people performing most tasks. 

Additionally, at the conclusion of each group’s work, a post-experiment assessment was taken 
from the students in that group regarding their experience throughout the session. In this evalu¬ 
ation, students answered a number of multiple choice teacher-designed feedback questions and 
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NASA TLX Subscale 

Description 

Mental Demand 

How much mental and perceptual activity was required? Was the task easy or demanding, 
simple or complex? 

Physical Demand 

How much physical activity was required? Was the task easy or demanding, slack or strenuous? 

Temporal Demand 

How much time pressure did you feel due to the pace at which the tasks or task elements 
occurred? Was the pace slow or rapid? 

Frustration 

How insecure, discouraged, irritated, stressed, and annoyed versus secure, gratified, content, 
relaxed, and complacent did you feel during the task? 

Effort 

How hard did you have to work (mentally and physically) to accomplish your level of 
performance? 

Own Performance 

How successful do you think you were in accomplishing the goals of the task set by the 
experimenter? How satisfied were you with your performance in accomplishing these goals? 

Table 2. Description of NASA TLX subscales. 


group-work evaluations (Angelo and Cross, 1993). Moreover, both sessions of the experiment were 
videotaped for post-analysis and further observations of students’ performance. 


DATA ANALYSIS AND EVALUATION 


Experiment Results 

Table 3 lists the results obtained from each session with regard to the 3 primary measures de¬ 
scribed in the previous Section. In this table, the average and coefficient of variance (CV) of each 
factor considering the performance of all 10 groups in each session are shown. The building volume 
(in cm 3 ) is calculated by multiplying the elevation of the topmost point on the building (in cm) by 
the building footprint (in cm 2 ). Also, the value of CV is calculated using Equation 1, 

CV = Standard Deviation + Mean Equation (1) 

As shown in this Table, the average building volume in the test group (that used the designed AR 
application for instruction and information delivery) was closer to the target value of 30,000 cm 3 . 



Building Volume (em3) 

Number of Elements 

Completion Time (min) 

Session 

Average 

CV 

Average 

CV 

Average 

CV 

1 (Control) 

34,801 

0.35652 

33 

0.18 

69 

0.10 

2 (Test) 

31,015 

0.15554 

29 

0.24 

73 

0.18 


Table 3. Primary measures statistics for control and test groups. 
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Also, the test group used fewer elements in their final design but spent slightly more time on the 
experiment. The difference in completion time was about 4 minutes on average which can be 
mostly attributed to the fact that students in this group had to spend some time upfront to learn 
how to use the AR application on their mobile devices to retrieve instructional videos and element 
information. It was also observed through the recorded video tapes that compared to the control 
group, students in the test group showed more involvement in the design process and spent larger 
portions of their experiment time on communication and exchanging ideas. 



Building Cost ($) 

Embodied Carbon (ton) 

Fire Resistance (hr) 

Session 

Average 

CV 

Average 

CV 

Average 

CV 

1 (Control) 

3,391,140 

0.479776 

3,665 

0.6095 

2.515 

0.2136 

2 (Test) 

4,412,160 

0.436926 

5,270 

0.6259 

2.361 

0.0751 


Table 4. Secondary measures statistics for control and test groups. 


Table 4 lists the results obtained from each session with regard to the 3 secondary measures 
described in the previous Section. In this table, the average and CV of each factor considering the 
performance of all 10 groups in each session are shown. 

As shown in this Table, the average building cost and embodied carbon is significantly less for 
the control group (session 1) than the test group (session 2). However, no statistical difference was 
observed between the average fire resistance factors for both groups, considering the CV values. 
The authors believe that one contributing factor to these results is that students in the test group 
had to scan building elements one by one and for as many times as needed during the experiment 
in order to retrieve information, while students in the control group had this information readily 
available in their print manuals during the entire time of the experiment. The need for the repetitive 
use of mobile devices to retrieve information may have caused frustration in the test group students. 
This problem coupled with the fact that students were under pressure to finish their designs on time 
might have ultimately resulted in less efficient secondary calculations in the test group. 

NASA TLX Results 

Figure 8 shows the results obtained from the control and test groups with respect to the 6 NASA 
TLX subscales. To statistically compare the results of the NASA TLX assessment technique, the p-value 
test was conducted. The p-value test is a statistical significance testing method which calculates the 
probability of obtaining a test statistic result at least as extreme as the one that was actually observed, 
with the assumption that the null hypothesis is true (Mendenhall and Sincich, 1991). Here, the null 
hypothesis was that “each NASA TLX subscale is statistically equal between the two treatments”. 
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Figure 8. Calculated NASA TLX subscales for control and test groups (out of 100). 


Results of the p-value analysis revealed that although students in the test group felt more frustrated 
(p-value = 0.044), they believed that they performed relatively better (p-value = 0.081). 

Additionally, according to the NASA TLX final assessment results, as shown in Figure 9, the average 
workload score achieved by both groups are almost the same. However, besides the time and effort 
students in both groups had to spend on the actual building design and assembly task, students in 
the test group had to spend extra time and effort to first learn how to work with the AR application 
to extract information. In other words, relative to the control group, the workload of students in the 
test group was divided between a main activity (i.e. building design and assembly) and an upfront 
activity (i.e. learning how to use an application). Therefore, it can be inferred that the test group stu¬ 
dents were under less workload as far as the actual building design and assembly task was concerned. 

Post-experiment assessment results 

Control treatment (Session 1) 

According to the results of the post-experiment assessment taken at the conclusion of the build¬ 
ing design and assembly task in session 1, 97% of respondents stated that the print manual instruc¬ 
tions about the overall goal and steps of the experiment were “very clear” or “clear”. Also, 79% of 
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respondents stated that it was “very easy” to retrieve primary and secondary measurement infor¬ 
mation from the print manual. Students were also asked to estimate the percentage of experiment 
time they spent on communicating with their team members. On average, 73% (standard deviation 
of 24%) of students' time was spent on communication and exchanging ideas. Moreover, students 
believed that on a scale of 0-100, the level of “interactivity” of the experiment was 86% (standard 
deviation of 17%). In order to evaluate the role of teamwork and collaboration on individual’s per¬ 
formance, each student in a team was asked to estimate the percentage of work he or she could 
have completed alone had he or she been given twice the time. The average response to this ques¬ 
tion was 90% (standard deviation of 14%). Finally, participants rated their overall assessment of the 
experiment on a scale of 1-5 (1=lowest, 5=highest) at 4 (standard deviation of 0.6). 

Test treatment (Session 2) 

According to the results of the post-experiment assessment taken at the conclusion of the build¬ 
ing design and assembly task in session 2, 79% of respondents stated that the instructions delivered 
through the mobile AR application were “very effective” or “effective” in helping them obtain nec¬ 
essary information during the experiment. Only 36% of respondents stated that it was “very easy” 
to retrieve primary and secondary measurement information using the mobile AR application while 
54% believed this required several rounds of trial and error. Students were also asked to estimate 
the percentage of experiment time they spent on communicating with their team members. On 
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average, 87% (standard deviation of 18%) of students’ time was spent on communication and ex¬ 
changing ideas. Moreover, 89% of students believed that the designed mobile AR application was 
“interactive”. In order to evaluate the role of teamwork and collaboration on individual’s performance, 
each student in a team was asked to estimate the percentage of work he or she could have com¬ 
pleted alone had he or she been given twice the time. The average response to this question was 
89% (standard deviation of 13%). In responding to another question, 92% of students stated that 
the designed mobile AR application has been “very helpful” or “somewhat useful” in their learning 
process and only 4% of respondents stated that they were “distracted” by this application during 
their learning experiment. Moreover, a solid majority of 86% had a positive view of the possibility 
of using mobile AR applications in other courses for the purpose of learning abstract and difficult- 
to-understand topics. Along the same line, on a scale of 1-5 (1 = lowest, 5=highest), with a mean of 4 
(standard deviation of 1.3), students expressed their willingness to recommend the designed mobile 
AR application (or a similar AR tool) to their schoolmates and instructors for use in other courses. 
Finally, participants rated their overall assessment of the experiment on a scale of 1-5 (1 = lowest, 
5=highest) at 4 (standard deviation of 0.9). Figure 10 shows the breakdown of student responses 
(on a Likert scale of 1-5) to two key questions with regard to the effectiveness of the virtual instruc¬ 
tor and the AR information delivery. 


SUMMARY AND KEY FINDINGS 


The main goal of this research was to design, implement, and systematically assess the pedagogical 
value of an AR-based instruction and information delivery tool to student learning in a large-scale 
classroom setting at a university level. For this purpose, a total of 60 undergraduate construction 
and civil engineering students participated in two separate (control and test) building design and 
assembly experimental treatments. Student performance data and perception was collected and 
analyzed in both treatments in an effort to assess the benefits of using a virtual instructor and 
information delivery through AR compared to traditional content delivery using print manuals. A 
total of 6 measures (3 primary measures and 3 secondary measures) were used to evaluate each 
group’s performance. Furthermore, the NASA TLX method was used to check students’ workload 
during the experiment and finally, evaluation forms were used to perform an individual evaluation 
of each student at the conclusion of the experiment. Key findings of this study are presented below: 

• In general, and considering the values calculated for the 6 primary and secondary measures 
(building volume, number of elements, completion time, building cost, embodied carbon, and 
fire resistance), both control and test groups showed a satisfactory performance. However, it 
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Figure 10. Student rating of the effectiveness of the (a) virtual instructor, and (b) AR 
information delivery platform, on a scale of 1-5 (1=lowest, 5=highest). 


was found that test group students generally did better with respect to the primary measures, 
while control group students performed better with respect to secondary measures. 

• Compared to the control group students, students in the test group spent an average of 4 
minutes more to complete their tasks which can be mainly attributed to the fact that they 


WINTER 2015 


19 






















ADVANCES IN ENGINEERING EDUCATION 

Content Delivery Using Augmented Reality to Enhance Students’ 
Performance in a Building Design and Assembly Project 


needed to learn how to use the AR application before they could proceed with the actual 
design and assembly task. 

• Analysis of post-experiment data revealed that introducing test group students to a new in¬ 
structional technology (AR) stimulated their interest and increased their involvement in the 
experiment. 

• According to the NASA TLX results, despite the fact that the students in the test group were 
more frustrated, they believed that they performed generally better, used the same mental 
and physical abilities as the control group, and were able to use their communication skills 
and exchange ideas more frequently. 

• At the conclusion of the experiment, test group students had very positive views about the 
possibility of using mobile AR applications in other courses for the purpose of learning abstract 
and difficult-to-understand topics. 


DISCUSSION AND FUTURE WORK 


Considering the results achieved from this experiment and the authors’ observations, the follow¬ 
ing discussion points and suggestions are presented to help guide future work in this area: 

• If students receive proper instructions and become more familiar with new technologies 
through prior training, they are more likely to perform better in comparison with those at¬ 
tending regular classroom sessions. It is imperative that this will ultimately motivate students 
to participate more in class activities, communicate effectively, and play an active role in their 
learning process. 

• A major advantage of using AR information delivery instead of the traditional paper-based 
method was that although it did not eliminate the need for a real classroom environment and 
teamwork participation, it helped students become more autonomous in their learning experi¬ 
ence by assisting them during class without the constant presence or intervention of a teacher. 

• Visual analysis of the recorded experiment videos showed that in the control session (paper- 
based treatment), students spent more time reading and browsing through their print manuals. 
Most of them started to talk to their teammates only once they had almost made up their minds 
about the solution. In contrast, in the test session (AR-based treatment), students started to 
talk to each other from the beginning of the session. Most of them watched the instructional 
videos together and tried to understand each step by explaining it to their teammates. Hence, 
from the post-video analysis, it was observed that students in the AR-based treatment spent 
more time on communication, were more engaged in the experiment, and played almost 


20 


WINTER 2015 





ADVANCES IN ENGINEERING EDUCATION 

Content Delivery Using Augmented Reality to Enhance Students’ 
Performance in a Building Design and Assembly Project 



equal roles in comparison to the paper-based treatment. Therefore, based on these observa¬ 
tions and the students’ self-report, it is evident that students in the AR-based treatment used 
their communication skills more frequently in comparison to the students participating in the 
paper-based treatment. 

• As previously mentioned, participating students in each treatment were separated into male- 
only and female-only groups to see if a noticeable difference would be observed. Gender- 
specific results indicated that in the first treatment (control group), in 5 (2 primary and all 3 
secondary) out of 6 (primary and secondary) factors, male groups did on average a better 
job in comparison to female groups. However, in the second treatment (test group), in 4 (1 
primary and all 3 secondary) out of 6 (primary and secondary) factors, the results were in 
favor of female groups, suggesting that female groups did a better job using the AR platform 
in comparison to male groups. Therefore, it can be concluded that female students performed 
better using the AR platform. 

The authors are currently working on expanding the application domain of this AR application 
and similar context-aware visual technologies to other engineering courses where the potentials 
and shortcomings can be better studied, and improvements to the existing teaching strategies can 
be made and tested in more diverse educational settings. 
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